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Summary of claimed subject matter 

The present invention takes the form of methods and apparatus for indexing the 
multimedia content of Web pages using conventional text-based Internet indexing and searching 
facilities. A Web page to be indexed is processed to identify media incorporated by reference 
into that Web page and to extract information describing the media data thus identified fi-om the 
referencing Web page, from the media file itself, and from other sources. The extracted 
information is then inserted as text annotations into a copy of the original Web page to produce 
an enhanced Web page which may then be processed in the normal way by conventional text- 
based bitemet indexing and searching facilities. 

Independent claims 1 , 6, 8, and 1 0 expressly define subject matter which the Examiner 
concedes is not disclosed by cited Nelson et al. Patent 6,243,713 (hereinafter "Nelson**). These 
claims are reproduced below with added citations to representative portions of Appellants' 
specification and drawings describing corresponding structure or acts. 

1 . Apparatoa for indexing a Web page [page 4, lines 4-S] which incorporates multimedia 
data \page 3, lines 24-27] by reference to one or more resources {page 4, lines 9-1 5; page 5, 
lines 4-1 SJ which supply said multimedia data, said method comprising, in combination: 

means for analyzing said web page to identify at least one markup tag containing a 
reference to a given one of said resources [Fig. J at 13; Fig. 2 at 35 and 38; page 2. lines 16-18; 
page 6. lines 13-20; page 7, line 13 to page 8, line J], 

means for selecting and executing a media processing program for analyzing the content 
of the multimedia data supplied by said given one of said resources to generate metadata 
describing said content {Fig 1 at 15, 16 and 1 7; Fig 2 at 40; page 2, line 25 to page 3. line 
5page 8. line 24 to page 13, line 23\, 

means for formatting said metadata into a character-based text annotation, {Fig. 1 at 20; 
Fig. 2 at 42; page 3, lines 6-8; page 11, lines 10-20] 

means for combining said Web page and said annotation to form an enhanced Web page, 
[Fig. 1 at 22; Fig. 2 at 46; page 2, lines 19-23; page 3, lines 8-10; and page 6, lines 2-5] and 
means for indexing said enhanced Web page. [Fig 1 at 25; Fig 2 at 50; page 2. lines 
21-22; page 3. lines 10-11; page 6. lines 5-10 and lines 22-23; page 6, line 25 to page 7. line 5; 
and page 13, line24 to page 14, line 7] 

2 
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4. (currently amended) Apparatus as set forth in claim 1 including means for acquiring 
additional metadata which describes the multimedia data supplied by said given one of said 
resources fix)m a source other than the content of said multimedia data, and means for including 
said additional metadata in said character-based text annotation, [page 2, line 19; page 3, lines 2- 
5; page 8, lines 1 1-23] 

6. Apparatus as set forth in claim 4 wherein said given resource is accessed through the 
operating system of a computer which provides said given resource and wherein at least some of 
said additional metadata includes information obtained from said operating system, [page 3, lines 
2-5; page 8, lines 19-23] 

8. Apparatus for collecting and storing metadata describing a hypertext Web page {page 
4, lines 4-6]^ said Web page including markup tags which identify multimedia data from one or 
more different external resources {page 4, lines 9-15; page 5, lines 4-13], said apparatus 
comprising, in combination, 

a parser for identifying said markup tags in said Web page, [page 7, line 13 to page 8, 

line 10] 

processing means for analyzing the content of said multimedia data identified by said 
markup tags to generate metadata describing said multimedia data, [Fig. 1 at 15, 16 and 17; Fig. 
2 at 40; page 2, line 25 to page 5, line Spage 8, line 24 to page 13, line 23] 

means for translating said metadata into a character-based text aimotation describing said 
multimedia data. [Fig. 1 at 20; Fig. 2 at 42; page i, lines 6-8; page 11, lines 10-20] and 

means for storing the combination of a copy of said Web page and said annotation to 
form an enhanced Web page suitable for processing by text-based indexing and searching 
facilities. [Fig. 1 at 22, 25; Fig. 2 at 46, 50; page 2, lines 19-23; page 3, lines 8-11; and page 6, 
lines 2-5, lines 5-10 and lines 22-23; page 6, line 25 to page 7, line 5; and page 13, line24 to 
page 14, line 7] 

1 0. The method of automatically enhancing the content of a Web page which 

contains multimedia data incorporated by reference \page 4, lines 9-15; page 5, lines 4- 

3 
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13], which comprises, in combtnation, the steps of: 

identifying one or more marlaip tags in said Web page which respectively identify 
one or more external resources which provide said multimedia data; [Fig. 1 at 13; Fig, 2 
at 35 and 38; page 2, lines 16-18; page 6, lines 13-20; page 7, line 13 to page 8, line J] 

generating metadata which describes said multimedia data» [Fig. 1 at J 5, 16 and 
1 7; Fig. 2 at 40; page 2, line 25 to page 3. line 5page 8, line 24 to page 13, line 23] 

translating said metadata into a character-based text annotation, [Fig, I at 20; Fig. 
2 at 42; page 3, lines 6-8; page 11, lines 10-20] and 

inserting said annotation into said Web page to form an enhanced Web page 
suitable for processing by a character-based text processing system. [Fig. 1 at 22, 25; 
Fig. 2 at 46, 50; page 2, lines 19-23; page 3, lines 8-11; and page 6, lines 2-5, lines 5-10 
and lines 22-23; page 6, line 25 to page 7, line 5; and page 13, line24 to page 14, line 7] 

Grounds of rejection to be reviewed on appeal 

In the Office Action mailed on April 26, 2005, independent claims 1, 8 and 10, 
and dependent claims 2, 4-7, and 1 1-17 were rejected as being directed to subject matter 
deemed to be obvious in view of Nelson. Claims 3 and 9 were rejected as being directed 
to subject matter deemed to be obvious over Nelson considered with Mohan et aL Patent 
6,748,382 (hereinafter ^TMohan"). The present appeal seeks review of the rejection of 
the three independent claims 1, 8 and 1 1 (and hence also the rejection of remaining 
dependent claims which include the limitations of the independent claims) . 

Ground 1 (Independent claims 1, 8 and 10): The Examiner concedes that 
Nelson does not disclose combining a Web page that incorporates multimedia data by 
reference with a text based annotation derived from and describing that multimedia data, 
thereby forming an enhanced Web page, but the Examiner contends that it would have 
been obvious to one of ordinary skill to modify the system described by Nelson to 
perform that claimed function. 

Ground 2 (Dependent claims 6): The Examiner concedes that Nelson does not 
teach acquiring at least some of the metadata describing multimedia data from the 
operating system that provides the multimedia data rather than from the content of the 

4 
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multimedia data, but the Examiner contends that it would have been obvious to one of 
ordinary skill to modify the system described by Nelson so that it performed that claimed 
function. 

Argument 

The applicable law 

The two grounds for rejection presented for review by this appeal are both based 
on the Examiner's conclusion that it would have been obvious to one of ordinary skill to 
modify the system described by Nelson to incorporate subject matter not disclosed by 
Nelson. The applicable law governing such rejections for obviousness is well stated in 
Sections 2143-2143.01 of the M.P.E.P. and relevant passages ftom those sections are 
reproduced below for ease of reference. 

"2143 Basic Requirements of a PHma Facie Case of Obvlonsness 

To establish a prima facie case of obviousness, three basic criteria must 
be met. First, there must be some suggestion or motivation, either in the 
references themselves or in the knowledge generally available to one of ordinary 
skill in the ait, to modify the reference or to combine reference teachings. Second, 
there must be a reasonable expectation of success. Finally, the prior art reference 
(or references when combined) must teach or suggest all the claim limitations. 

The teaching or suggestion to make the claimed combination and the 
reasonable expectation of success must both be found in the prior art, and not 
based on applicant's disclosure. In re Vaeck, 947 F.2d 488, 20 USPQ2d 1438 
(Fed. Cir. 1991)." [M.P.E.P. §2143] 

"2143.01 Suggestion or Motivation To Modify tiie References [R-2] 

THE PRIOR ART MUST SUGGEST THE 
DESIRABILITY OF THE CLAIMED INVENTION 

There are three possible sources for a motivation to combine references: 

the nature of the problem to be solved, the teachings of the prior art, and the 

knowledge of persons of ordinary skill in the art." In re Rouffet, 149 F.3d 1350. 

1357, 47 USPQ2d 1453, 1457-58 (Fed. Cir. 1998) (The combination of the 
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references taught every element of the claimed invention, however without a 
motivation to combine, a rejection based on a prima facie case of obvious was 
held improper,). The level of skill in the art cannot be relied upon to provide the 
suggestion to combine references. Al-Site Corp. v. VSIInt 7 Inc., 174 F.3d 1308, 
50 USPQ2d 1161 (Fed. Cir. 1999)." 

FACT THAT REFERENCES CAN BE COMBINED OR MODIFEED 
IS NOT SUFFICIENT TO ESTABLISH PRIMA FACIE OBVIOUSNESS 

The mere fact that references can be combined or modified does not 

render the resultant combination obvious unless the prior art also suggests the 

desirabiUty of the combination. In re Mills, 916 F.2d 680, 16 USPQ2d 1430 (Fed. 

Cir. 1990) * * * Although a prior art device "may be capable of being modified 

to run the way the apparatus is claimed, there must be a suggestion or motivation 

in the reference to do so." 916 F.2d at 682, 16 USPQ2d at 1432.). See also In re 

Fritck, 972 F.2d 1260, 23 USPQ2d 1780 (Fed. Cir. 1992) 

THE PROPOSED MODIFICATION CANNOT RENDER 

THE PRIOR ART UNSATISFACTORY FOR ITS INTENDED PURPOSE 

If proposed modification would render the prior art invention being 
modified unsatisfactory for its intended purpose, then there is no suggestion or 
motivation to make the proposed modification. In re Gordon, 733 F.2d 900, 221 
USPQ 1125 (Fed. Cir. 1984)" [M.P.E.P. §2143.01] 

Ground 1 (Independent Claims 1» 8 and 10) 

In the Office Action mailed on August 30, 2004, at page 4, the Examiner explained his 

rejection of the independent claims in view of Nelson as follows: 

"Nelson does not teach combining the Web page and the generated 
annotation (Nelson calls it a token) to form an enhanced Web page. As stated 
earlier. Nelson does teach that all of the tokens extracted from a particular 
document are linked to that document through use of a unique document ID in 
col. 25 lines 24-36. Therefore, tokens of text, image, video, audio, and other data 
are all linked together to represent a single document through use of the unique 
document ID. Thus, this is essentially the same in function, but only similar in 
form to the claimed limitation. It would have been obvious to one of ordinary skill 

6 
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in the art at the time the invention was made to have modified Nelson to have 
created the claimed invention. It would have been obvious and desirable to have 
modified Nelson to have inserted the token representations of image, video, audio, 
and other multimedia components into the text of the multimedia document so that 
the text did not have to be processed, thus allowing the extraction software design 
to be simpler. 

In the response mailed on January 31, 2005, at page 7, Appellants' advanced three 
reasons why the modification of Nelson proposed by the Examiner would not have been obvious 
as follows; 

"(1). The suggested modification would not allow the extraction software to be 
simpler as suggested by the Examiner. The step of extracting descriptive character based 
tokens (e.g. the speech recognition step for audio media data) would need to be 
performed either way, and placing the tokens into the original document would not 
eliminate or simplify the token extraction software; 

(2) In Nelson's system, the tokens need to be placed into the index to permit 
retrieval, but there is no need in Nelson's system to also place the tokens in the original 
Web page, and to do so would require that the Web page be rewritten and the resulting 
annotated Web page would consume more space, without serving any useful purpose. 

(3) Applicants' invention permits Web pages to indexed and searched by existing 
automated search engines C*Web crawlers" or "spiders'") such Google, Yahoo, etc., 
whereas Nelson teaches the use of proprietary indexing and search scheme. Applicant's 
invention thus (a) completely eluninates the need to design and deploy a new indexing 
and search system and (b) makes the Web pages that are automatically annotated using 
applicants' invention searchable at by using existing popular Web search facihties. 

It is submitted that there is nothing in the disclosure of Nelson that suggests 
applicants' invention as claimed, and one of ordinary skill in the art would have no 
reason (absent applicants' disclosure) to modify the Nelson indexing and search system 
so that it annotated the original Web pages being indexed with metadata extracted from 
referenced media data." 

In the final rejection mailed on April 26, 2005, the Examiner did not dispute the 
arguments above that demonstrate why one skilled in the art would not be motivated to make the 
proposed modification in order "to allow the extraction software to be simpler" as stated earlier. 
Instead, in his "Response to Arguments" at page 1 1 of the final rejection, the Examiner advanced 
the following new basis for contending that the proposed modification of Nelson would have 
been obvious: 

"Nelson parses the media content and generates tokens describing the 
media content and associates the tokens with the parent document as is described 
in fig. 8-9 and col. 14 line 56 - col 25 line 58. The difference is that the tokens 
are maintained separately and have an associative link to the document from 

7 
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which they were created. The documents which Nelson parses and indexes can be 
HTML web documents. The HTML specification provides for meta-tag annotation 
of the document to provide existing popular Web search facilities with a way to 
automatically index such documents. Since Nelson teaches how to extract 
character-based annotation data from the multimedia components of the web 
page, the Examiner believes it would have been obvious to have recombined the 
extracted character data with the Web page using the HTML annotation meta- 
tags capability. Thus, the Examiner maintains the rejection of the claimed 
invention as being obvious over the teachings of Nelson. " 

While the HTML specifications, cited at page 4, lines 16-23 of Appellants' specification, 
do •'provide for meta-tag annotation of the document to provide existing popular Web search 
facilities with a way to automatically index such documents" as noted by the Examiner, the 
Examiner points to nothing in the HTML specifications that suggests that character-based 
annotation data extracted fi:om the multimedia components of a Web page should be inserted into 
that Web page to form an enhanced Web page as claimed. That suggestion is foimd in only in 
Appellants' disclosure. 

Nelson plainly fails to suggest that conventional text-based indexing systems could be or 
should be used to index the multimedia data imbedded by reference in Web pages. Indeed, 
Nelson instead teaches that conventional text-based indexing systems are inadequate for 
indexing multimedia data, and teaches instead that the Nelson multimedia indexing system 
should be employed instead to handle Web pages that include multimedia data. Since clearly 
Nelson **teaches away** firom using conventional text based indexing to handle compoimd Web 
pages that incorporate multimedia, and since there is no suggestion at all in the HTML 
specification that meta tags can or should be used to describe a Web page's multimedia 
components, there is simply nothing in the cited prior art relied upon by the Examiner that 
suggests, or would motivate one skilled in art to attempt, the modification of the Nelson system 
that the Examiner asserts to be obvious. 

There is notiiing in the HTML specifications that would motivate one skilled in the art to 
modify the Nelson system to rewrite Web pages to include metadata describing a page's media 
data components. If for some reason (not foimd in the prior art) one skilled in the art might 
consider such a modification, there are only two possible ways the modified Nelson indexing 
system might work: Either the metadata extracted from the media data would be placed in the 
original Web page (1) instead of being placed in Nelson's inverted file multimedia index, or (2) 

8 
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the metadata would be placed in the Web page in addition to being placed in Nelson's 
multimedia index. But a closer look reveals that neither modification would make engineering 
sense. 

The first approach, placing the extracted multimedia metadata into the Web page as meta- 
tags insteadof placing it into Nelson's inverted file multimedia index, would render the Nelson 
system inoperable. The robust functions which the Nelson system perfoms simply won*t work 
unless the extracted metadata is placed in the disclosed multimedia index structure which Nelson 
uses to respond to compoimd multimedia queries. Since modifying Nelson to place the metadata 
into the original Web page instead of into the structured index would render the Nelson system 
unsatisfactory for its intended purpose, it plainly cannot be said that one skilled in the art would 
be motivated to modify the Nelson system in that way. In re Gordon, 733 F,2d 900, 221 USPQ 
1125 (Fed. Cir. 1984) 

One skilled in the art would Ukewise not be motivated try the second approach; that is, to 
modify the Nelson system so that it stored the extracted metadata in both the original Web page 
and in the structured index. Because the Nelson system can more rapidly perform any needed 
indexed search using its structured index as disclosed by itself, there is shnply no useful purpose 
to be served in also rewriting the original Web pages, particularly in view of the fact that the 
annotated Web pages would consume more storage space. The Nelson system is capable of 
performing all of the search functions that text-based indexing system can perform, and further 
has the ability to perform special purpose query functions that cannot be performed using 
conventional text based indexing systems. For example, because Nelson's structured index 
includes not only tokens but also infomiation about the position of multimedia components 
within each Web page. Nelson's structured index makes it possible to perform additional 
positional compound queries (for example, a search for the word "beach" within 10 words of an 
image of a sunset as described by Nelson at col. 3, lines 9-12). One of ordinary skill would thus 
not be motivated to also place in the original Web page and then index that enhanced Web page 
with conventional text-based indexing systems since, as Nelson teaches, those conventional 
systems do nothing that Nelson's indexing system can*t do, have shortcomings that Nelson's 
"unified index" system is said to overcome, and require the larger rewritten Web pages that take 
up more space. 

9 
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Ground 2 (Dependent claims 6) 

While the Examiner concedes that Nelson does not teach the subject matter recited in 
claim 6, acquiring at least some of the metadata describing multimedia data from the operating 
system that provides the multimedia data rather than from the content of the multimedia data, the 
Examiner contends that it would have been obvious to one of ordinary skill to modify the system 
described by Nelson so that it performed that claimed function. At page 12 of the final rejection, 
the Examiner states "the Examiner believes it would have been advantageous to have acquired 
data through the operating system for components dependent on particular hardware 
configurations and thus believes Nelson would have been modified to specifically acquire such 
metadata." 

But nothing in the cited references suggests that metadata about multimedia components 
could or should be acquired from the operating system. Appellants agree of course that it 
advantageous to do that. The operating system's file system, for example, contains information 
about the size and creation dates of the referenced multimedia files, and this kind of metadata 
would be advantageous to include in the enhanced Web page. But Nelson does not teach getting 
metadata from the operating system, and that fact, coupled with the fact that '"it would have been 
advantageous" to do so as stated by the Examiner, suggests that doing so was not obvious to 
Nelson. The rejection of claim 6 for obviousness is not justified by any teaching or suggestion in 
the prior art, and the Examiner has pointed to no such teaching. The rejection of claim 6 should 
be reversed. 

Conclusion 

There is no hint or suggestion of Appellants' invention as claimed anywhere in the 

Nelson teaching, or in the HTML specification. Neither Nelson nor the HTML specification 

suggests extracting metadata describing the multimedia components referred to in a Web page 

and placing that metadata back into that Web page to yield an enhanced Web page that can be 

searched by conventional Internet search facilities. One skilled in the art would not be motivated 

by any teaching in the prior art to modify the Nelson system so that it created enhanced Web 

pages since such a modification (depending on how it was attempted) would either disable the 

Nelson System as disclosed or would only bulk-up the indexed Web pages without serving any 

useful purpose. Likewise, there is not teaching or suggestion in the prior art that metadata 

10 
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describing multimedia Web page components can be usefully obtained from the operating 
system that provides access to those components as set forth in claim 6. 

It is accordingly submitted that the Examiner has to establish that the invention set forth 
in independent claims 1, 8 and 10 and in dependent claim 6 would have been obvious to one of 
ordinary skill in the art. The Examiner's fmal rejection of claims I, 8 and 10> and of claim 6, as 
being obvious in view of Nelson should be reversed. 



Claims appendix 

An appendix containing a copy of the claims involved in the appeal is attached. 

Respectfiilly submitted. 
Dated: September 24, 2005 Charles G. Call, Reg. 20,406 

Certificate of Transmission under 37 CFR 1>8 

I hereby certify that this Appellants ' Brief is being transmitted by facsimile to the central 
facsimile number of the U.S. Patent and Trademark Office, (703) 872-9306, on September 24, 
2005 

Dated: September 24, 2005 Signature . 




Charles G. Call, Reg. No. 20,406 
USPTO Customer No. 021253 
68 Horse Pond Road 
West Yarmouth, MA 02673 
Ph. (508) 778-2630 Fax (508) 629-6540 
call(5^ateatsoft.com 
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CLAIMS APPENDIX 

1. Apparatus for indexing a Web page which incorporates multimedia data by reference 
to one or more resources which supply said multimedia data, said method comprising, in 
combination: 

means for analyzing said web page to identify at least one markup tag containing a 
reference to a given one of said resources, 

means for selecting and executing a media processing program for analyzing the content 
of the multimedia data supplied by said given one of said resources to generate metadata 
describing said content, 

means for formatting said metadata into a character-based text annotation, 

means for combining said Web page and said annotation to form an enhanced Web page, 

and 

means for indexing said enhanced Web page. 

2. Apparatus as set forth in claim 1 wherein said means for selecting and executing a 
media processing program comprises means for determining the particular data type of the 
multimedia data supplied by said given resource and means for selecting a processing program 
for analyzing multimedia data formatted in accordance with said particular data type. 

3. Apparatus as set forth in claim 1 wherein said means for formatting said metadata 
comprises means for generating a text data annotation expressed in accordance with the 
Extensible Markup Language. 

4. Apparatus as set forth in claim 1 including means for acquiring additional metadata 
which describes the multimedia data supplied by said given one of said resources fiom a source 
other than the content of said multimedia data, and means for including said additional metadata 
in said character-based text annotation. 

5. Apparatus as set forth in claim 4 wherein at least some of said additional metadata 
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includes infonnation obtained from said one markup tag. 

6. Apparatus as set forth in claim 4 wherein said given resource is accessed through the 
operating system of a computer which provides said given resource and wherein at least some of 
said additional metadata includes information obtained from said operating system. 

7. Apparatus as set forth in claim 4 wherein at least some of said additional is obtained 
via the Internet. 

8. Apparatus for collecting and storing metadata describing a hypertext Web page, said 
Web page including markup tags which identify multimedia data from one or more different 
external resources, said apparatus comprising, in combination, 

a parser for identifying said markup tags in said Web page, 

processing means for analyzing the content of said multimedia data identified by said 
markup tags to generate metadata describing said multimedia data, 

means for translating said metadata into a character-based text annotation describing said 

multimedia data, and 

means for storing the combination of a copy of said Web page and said annotation to 
form an enhanced Web page suitable for processing by text-based indexing and searching 
facilities, 

9. Apparatus as set forth in claims 8 wherein said text annotation is expressed in the 
Extensible Markup Language. 
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10. The method of automatically enhancing the content of a Web page which contains 
multimedia data incorporated by reference which comprises, in combination, the steps of: 

identifying one or more markup tags in said Web page which respectively identify one or 

more external resources which provide said multimedia data; 

generating metadata which describes said multimedia data, 

translating said metadata into a character-based text annotation, and 

inserting said annotation into said Web page to form an enhanced Web page suitable for 

processing by a character-based text processing system. 

1 1 . The method of automatically enhancing the content of a Web page as set forth in 
claim 10 wherein said step of identifying one or more markup tags comprises the steps of first 
identifying markup tags in said Web page and extracting the uniform resource locator (TJRL) of 
one of said external resources from at least selected ones of said markup tags. 

12. The method of automatically enhancing the content of a Web page which contains 
multimedia data as set forth in claim 10 wherein said step of generating metadata includes the 
sub-steps of retrieving said multimedia data from said one or more external resources and 
analyzing the content of said multimedia data to extract said metadata therefrom. 

13. The method of automatically enhancing the content of a Web page as set forth in 
claim 12 wherein said step of generating metadata comprises the sub-steps of identifying the data 
type of the multimedia data from each of said resources and then selecting a processing routine 
for multimedia of the identified data type from each of said resources. 

14. The method of automatically enhancing the content of a Web page as set forth in 
claim 10 includes the further step of indexing said enhanced Web page to provide access to said 
Web page in response to queries expressing one or more attributes expressed in said text 
annotation. 



14 



PACE 16/17 " RCVD AT 9/24/2005 9:31:11 PM [Eastern Daylight Time) " SVR:USPTO-EFXRF-6/24 * DN1S:2738300 ' CSID:508 778 2830 * DURATION (mm-ss):07-34. 



Sep 24 2005 9:31PM CHHRLES G CHLL 



508-778-2630 



p- 17 



15, The method of automatically enhancing the content of a Web page as set forth in 
claiin 10 includes the further step of searching the content of said enhanced Web page in 
response to a search request to determine if attributes expressed in said search request are 
contained in said text annotation. 

16, The method of automatically enhancing the content of a Web page as set forth in 
claim 13 includes the further step of indexing said enhanced Web page to provide access to said 
Web page in response to queries expressing one or more attributes expressed in said text 
annotation. 

17, The method of automatically enhancing the content of a Web page as set forth in 
claim 13 includes the further step of searching the content of said enhanced Web page in 
response to a search request to determine if attributes expressed in said search request are 
contained in said text annotation. 
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